Repository navigation
Conversation
|
Are we ready to remove |
|
Yeah I think this will replace |
adamnovak
left a comment
There was a problem hiding this comment.
I found a few problems with this.
- There aren't any tests. There need to be tests for different edge cases of path handling (paths that overlap between input graphs, paths with gaps between them, paths that use phase blocks and don't really want to be connected, paths that are in pieces in their original input graphs) so that we can ensure that the code actually handles paths in the way we expect and produces the appropriate errors when graphs shouldn't be able to be combined.
- Because there aren't any tests, I think the code is able to do the wrong things with some of those situations. I think it's able to produce paths where their subranges don't actually reflect the total length of all their visits, for example. It's also claiming to be able to change the sense of paths from HAPLOTYPE to REFERENCE.
- The comments often substitute a summary of the code for an explanation of the abstraction that the code is supposed to provide to its caller to work with. This includes the user-facing documentation for the options.
- In general the different modes don't share code where it seems like they should, and don't explain why they can't. It looks like
-fmode was implemented to combine graphs and join up paths, and then-mmode was implemented to combine fragments along a reference-sense backbone and join up paths, and besides the path-joining function they share basically nothing, with similar problems solved by main force with completely different code. I would expect to see one main implementation of loading and combining the graphs, which is able to do different things at different points by e.g. checking a condition and deciding whether or not to enforce constraints around a single REFERENCE set of paths. I'm not sure the reference mode can't just be a pass, rather than its own nearly-complete implementation. - We're approaching the level of complexity where we no longer want all the code to live in the subcommand file. There probably should be a widget class in
srcor some new algorithms insrc/algorithmsto hold the guts of this.
|
I've taken care of removing |
|
I did a quick check through the old issues, and this may be relevant to #2688? |
|
Yeah, this should fix #2688. |
|
Ah, this line is going to need to be replaced by Line 149 in d9ea751 |
|
#2677 may also be relevant, if this PR means we could do the workflow Adam said was impossible here: #2677 (comment) |
adamnovak
left a comment
There was a problem hiding this comment.
I think this will work, and is possible to understand.
I also think the core abstractions could stand to be redesigned and, especially, more clearly explained. What's an accumulator? What's the general idea of the approach that a person will need to know to understand what the methods are for? Can we avoid weird states like an un-described Chunk?
The tests are thorough, but grouped oddly, and they are probably too thorough in easy areas while missing some cases that are actually important. We have 105 new command-line tests, many of which seem to be testing the same basic questions as unit tests. Tests are there for a lot of the error message strings, but verify the wording of the string rather than the presence of the actual path name/node number/etc. There are unit tests for input validation functions but only for valid and not invalid inputs.
I only gave the core implementation code a cursory scan.
I'm putting this in as a comment because I don't know if it makes sense for you to actually try and do all this cleanup @xinzilan. We want to merge this and it seems like it will, plausibly, work, and the structure isn't so terrible that it couldn't be maintained. I think you'd learn a lot form making all the changes I recommend, but most of the issues are the sort of issues that come out of a code generator. You probably could set the code generator to fixing them and get rid of most of them, but what I really want to convey is that it would be useful if you learned to spot these sorts of things and shake them out of code before submission.
If it were me, I would re-comment the headers, cut almost all the CLI tests, maybe make a few of the refactoring changes if it's clear that those give a more coherent final product, and maybe leave the rest alone? If you want to do a really good job, you can define some custom exception types for all the error messages, move some of the hundred CLI tests about error messages over into C++ unit tests about exception messages instead, and keep just the CLI tests for things about the CLI that the unit tests can't get at usefully.
But if @benedictpaten thinks we need the graph merging as soon as possible (which we do), we can also just merge this and postpone that sort of cleanup until later/never.
| } | ||
| logger.error() << "unrecognized --seam policy \"" << value | ||
| << "\"; expected one of: renumber, shared, trim." << endl; | ||
| return SeamPolicy::RENUMBER; // not reached; logger.error() exits |
There was a problem hiding this comment.
It would be nice if the comment didn't have to contradict the code like this.
We don't have std::unreachable() yet.
We should mark the destructor for the error stream [[noreturn]], but I don't think the compiler is guaranteed to be able to figure out that that means subsequent statements are unreachable. Even if it could, right now we're using one type with a flag for whether to exit, so we'd have to refactor it to be a special type for fatal errors. (See
Lines 60 to 64 in d08602f
I would do something like:
| return SeamPolicy::RENUMBER; // not reached; logger.error() exits | |
| throw std::logic_error("unreachable"); // logger.error() stream exits after logging |
There was a problem hiding this comment.
Done: I replaced the return with the throw std::logic_error("unreachable") you suggested.
| return chunk; | ||
| } | ||
|
|
||
| /// Create a path with full metadata and walk it over the given handles. |
There was a problem hiding this comment.
Walking the path would mean following it after it was created, not creating it so that it goes a certain way. You could maybe say "run it over the given handles", but that's also kind of confusing.
I would try:
| /// Create a path with full metadata and walk it over the given handles. | |
| /// Create a path with the given metadata, that visits the given steps. |
There was a problem hiding this comment.
Done: the comment now says it creates a path that visits the given steps.
| return path; | ||
| } | ||
|
|
||
| /// Create a REFERENCE-sense path, the sense --seam trim positions a chunk by. |
There was a problem hiding this comment.
This is part of the unit test, so it shouldn't really be cross-referencing the command line flags for vg combine. There's an abstraction barrier between the unit tests and the CLI.
| /// Create a REFERENCE-sense path, the sense --seam trim positions a chunk by. | |
| /// Create a REFERENCE-sense path. |
There was a problem hiding this comment.
Done: the comment no longer mentions the --seam trim flag.
| return names; | ||
| } | ||
|
|
||
| TEST_CASE("compute_path_length adds up what a path visits", "[combine]") { |
There was a problem hiding this comment.
But nodes themselves can't be summed.
| TEST_CASE("compute_path_length adds up what a path visits", "[combine]") { | |
| TEST_CASE("compute_path_length adds up visited node lengths", "[combine]") { |
There was a problem hiding this comment.
Done: I used your wording, and the function moved to libhandlegraph as you suggested.
There was a problem hiding this comment.
Done: I used your wording, and the function moved to libhandlegraph as you suggested in the following comment.
| vg convert -p $C/fb.gfa > $C/fb.pg 2>/dev/null | ||
| vg combine $C/whole.vg $C/fa.pg $C/fb.pg 2>$C/collide.err >/dev/null | ||
| is "$?" "1" "a merged name that is already taken is rejected" | ||
| is "$(grep -c 'already an unrelated path' $C/collide.err)" "1" "the error names the path standing in the way" |
There was a problem hiding this comment.
This isn't actually checking that the path name appears in the message.
There was a problem hiding this comment.
This error can't happen anymore as circularity is out of the key: a path with the merged name always has the same key as the fragments, so it is one of them. I replaced the error with an internal assertion and removed the test.
| vg construct -r small/x.fa -v small/x.vcf.gz > x.vg | ||
| vg construct -r small/x.fa -v small/x.vcf.gz | vg paths -d -v - > y.vg | ||
| vg construct -r small/x.fa -v small/x.vcf.gz | vg paths -d -v - > z.vg | ||
| C=combine |
There was a problem hiding this comment.
We probably should explain why we would store a one-word value in a one-letter variable. It looks like we're trying to keep our test scratch work in this directory?
There was a problem hiding this comment.
Done: yes, it's the scratch directory for the test's files. I renamed it to SCRATCH and added a comment saying so.
| printf "$RS"'S\t5\tGGTT\nS\t6\tA\nS\t7\tC\nS\t8\tGGGGG\nL\t5\t+\t6\t+\t0M\nL\t5\t+\t7\t+\t0M\nL\t6\t+\t8\t+\t0M\nL\t7\t+\t8\t+\t0M\nP\tGRCh38#0#chr1[8-18]\t5+,6+,8+\t*\nP\tHG002#1#chr1#0[8-18]\t5+,7+,8+\t*\nP\tHG002#2#chr1#0[12-18]\t6+,8+\t*\n' > $C/trb.gfa | ||
| printf "$RS"'S\t1\tAAAA\nS\t2\tC\nS\t3\tG\nS\t4\tGGGGG\nL\t1\t+\t2\t+\t0M\nL\t1\t+\t3\t+\t0M\nL\t2\t+\t4\t+\t0M\nL\t3\t+\t4\t+\t0M\nP\tGRCh38#0#chr1[0-10]\t1+,2+,4+\t*\nP\tHG002#1#chr1#0[0-10]\t1+,2+,4+\t*\nP\tHG002#2#chr1#0[0-5]\t1+,3+\t*\n' > $C/tlb.gfa |
There was a problem hiding this comment.
If we have to inline GFA in the tests, I would replace these with heredocs that someone can edit and read.
There was a problem hiding this comment.
Done: the inline GFA is all in heredocs now, one line per record. Fields are separated by spaces there, and tr turns them into tabs, so editors can't mix up tabs and spaces.
| vg combine -s trim -u $C/bl_dangle.pg $C/br.pg >/dev/null 2>$C/fuse.err | ||
| is "$?" "1" "--fuse rejects a boundary node with an edge on the welded side" | ||
| is "$(grep -c 'weld would have to drop' $C/fuse.err)" "1" "the error says what the weld would drop" |
There was a problem hiding this comment.
Are some of these tests overlapping with the unit tests? We can test everything as a unit test and again as a command-line test, but we're adding 105 command line tests here, which seems like enough that we might have to start to think about total runtime, and it might be best to focus on the things that we think still could possibly be wrong after the unit tests pass. If there was a good way to turn some of these into unit tests instead, and we could find a way to verify error messages from Catch tests (maybe by throwing exceptions instead of using the logger and then logging from a catch block?), things might be more manageable.
Maybe it is in fact better to constrain all important aspects of the program's behavior with tests? But if we want tests at that scale we might need to adopt something other than Bash-TAP to manage them, because Bash-TAP doesn't let us do things like run a particular failing test out of our 115 distinct tests.
There was a problem hiding this comment.
Done: the errors are exceptions now, so I moved the error cases into unit tests and dropped the command line tests that only re-checked what the unit tests cover. The 45 that are left check what the unit tests can't: option parsing, each mode end to end, an error from the combining code reaching the command line, and graphs made by other vg subcommands.
| # Whatever VPKG can load, combine can merge. | ||
| vg convert -p $C/adj1.vg > $C/adj1.pg | ||
| vg convert -a $C/adj2.vg > $C/adj2.hg | ||
| vg combine $C/adj1.pg $C/adj2.hg > $C/o_fmt.vg 2>/dev/null | ||
| is "$(vg paths -Ev $C/o_fmt.vg)" "$(vg paths -Ev $C/o_real.vg)" "PackedGraph and HashGraph inputs merge the same way" | ||
| is "$(vg stats -N $C/o_fmt.vg)" "32" "and keep the same nodes" |
There was a problem hiding this comment.
If we really want to test that, we need to try some of the non-mutable graphs (.xg, .gbz).
If we're just going to test that it works when the graph is a HashGraph and when it is a PackedGraph, I don't think it's worth testing at all.
There was a problem hiding this comment.
Done: I removed it. Combining needs mutable graphs, so read-only formats like GBZ are rejected, and there's already a test for that.
adamnovak
left a comment
There was a problem hiding this comment.
I think this will work, and is possible to understand.
I also think the core abstractions could stand to be redesigned and, especially, more clearly explained. What's an accumulator? What's the general idea of the approach that a person will need to know to understand what the methods are for? Can we avoid weird states like an un-described Chunk?
The tests are thorough, but grouped oddly, and they are probably too thorough in easy areas while missing some cases that are actually important. We have 105 new command-line tests, many of which seem to be testing the same basic questions as unit tests. Tests are there for a lot of the error message strings, but verify the wording of the string rather than the presence of the actual path name/node number/etc. There are unit tests for input validation functions but only for valid and not invalid inputs.
I only gave the core implementation code a cursory scan.
I'm putting this in as a comment because I don't know if it makes sense for you to actually try and do all this cleanup @xinzilan. We want to merge this and it seems like it will, plausibly, work, and the structure isn't so terrible that it couldn't be maintained. I think you'd learn a lot form making all the changes I recommend, but most of the issues are the sort of issues that come out of a code generator. You probably could set the code generator to fixing them and get rid of most of them, but what I really want to convey is that it would be useful if you learned to spot these sorts of things and shake them out of code before submission.
If it were me, I would re-comment the headers, cut almost all the CLI tests, maybe make a few of the refactoring changes if it's clear that those give a more coherent final product, and maybe leave the rest alone? If you want to do a really good job, you can define some custom exception types for all the error messages, move some of the hundred CLI tests about error messages over into C++ unit tests about exception messages instead, and keep just the CLI tests for things about the CLI that the unit tests can't get at usefully.
But if @benedictpaten thinks we need the graph merging as soon as possible (which we do), we can also just merge this and postpone that sort of cleanup until later/never.
# Conflicts: # autocomp.bash # autocomplete.sh # doc/vgmanmd.desc.md
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
adamnovak
left a comment
There was a problem hiding this comment.
I didn't make it fully through this but I found some things to change.
I'm also attaching an attempt at an automated review of the full PR using #5103. I haven't read it yet, but we could use it to evaluate that PR or maybe mine it for ideas.
| return clean; | ||
| } | ||
|
|
||
| /// Fuse two nodes into one node. |
There was a problem hiding this comment.
We might not want to duplicate doc comments on definitions.
| } catch (const std::invalid_argument& e) { | ||
| logger.error() << e.what() << endl; | ||
| } |
There was a problem hiding this comment.
This is meant to catch the not-enough-inputs signal from the combiner, but it's pretty general, so if something goes really wrong inside the combiner we might hit this instead of getting a stack trace. (Assuming our stack trace on exception stuff actually works...)
Maybe the CLI layer should be responsible for handling not having enough inputs (so it can print a useful message, and so we have the option to add some kind of override flag later in case we want to combine zero or more things in scripts), and the combiner should do something sensible with one input (pass it through) and zero inputs (produce an empty graph).
| /// | ||
| /// GraphCombiner::combine() is covered end to end by test/t/51_vg_combine.t. | ||
| /// These tests drive the steps underneath it, whose intermediate results -- the ID | ||
| /// shifts, the trimmed boundary, the merged coordinates -- the command line cannot see. | ||
| /// | ||
| /// Error branches are absent on purpose: logger.error() exits the process, which would | ||
| /// take the whole test binary with it. They live in test/t/51_vg_combine.t. |
There was a problem hiding this comment.
This isn't really that useful here. This is always the philosophy with the unit test files. If we have to reiterate it, we should do that more simply.
If we really want to test error logging we could adjust the logging system to let us capture errors during testing. Maybe we make it throw some kind of LoggedFatalError. Maybe we do that all the time and we catch and do the actual output somewheew in main()? But we don't need to do that to merge this.
There was a problem hiding this comment.
This is more tests than I am going to read. There hopefully aren't a lot of problems lurking in here.
| int64_t max_node_id = 0; ///< Largest node ID used in dest so far. | ||
| std::map<BasePathKey, SeamState> seams; ///< For each base path, the node, orientation, offset and input it ends at so far. | ||
|
|
||
| friend class unittest::TestGraphCombiner; |
There was a problem hiding this comment.
Rather than forward-declaring and being friends with the test class that exposes all this stuff, can't we make it protected instead of private so the test class can expose it? If it constitutes a unit that can need testing, it probably isn't an integral enough part of the containing class's unit to be private.
| tr ' ' '\t' > $SCRATCH/ml.gfa <<'EOF' | ||
| H VN:Z:1.1 RS:Z:GRCh38 | ||
| S 1 AAAA | ||
| S 2 C | ||
| S 3 G | ||
| S 4 GGGGG | ||
| L 1 + 2 + 0M | ||
| L 1 + 3 + 0M | ||
| L 2 + 4 + 0M | ||
| L 3 + 4 + 0M | ||
| P GRCh38#0#chr1[0-10] 1+,2+,4+ * | ||
| P HG002#1#chr1#0[0-10] 1+,2+,4+ * | ||
| P HG002#2#chr1#0[0-10] 1+,3+,4+ * | ||
| P HG005#1#chr1#0[0-10] 1+,3+,4+ * | ||
| EOF | ||
| # Right chunk including only HG002 | ||
| tr ' ' '\t' > $SCRATCH/mr.gfa <<'EOF' | ||
| H VN:Z:1.1 RS:Z:GRCh38 | ||
| S 5 TTTT | ||
| S 6 A | ||
| S 7 C | ||
| S 8 GGGGG | ||
| L 5 + 6 + 0M | ||
| L 5 + 7 + 0M | ||
| L 6 + 8 + 0M | ||
| L 7 + 8 + 0M | ||
| P GRCh38#0#chr1[10-20] 5+,6+,8+ * | ||
| P HG002#1#chr1#0[10-20] 5+,7+,8+ * | ||
| P HG002#2#chr1#0[10-20] 5+,6+,8+ * | ||
| EOF |
There was a problem hiding this comment.
These are legible, but is this really better than committing file fixtures?
| bool is_circular = false; ///< Whether this base path is circular. | ||
| }; | ||
|
|
||
| /// An input's reference path: its name, its base path, its first and last nodes, where it starts and ends on that base path, and whether it is circular. |
There was a problem hiding this comment.
We can't list more than like 2 things in a class description like this; we need to explain what the class represents. We already have the list of fields in the body. I guess this is for holding metadata about a reference path as it exists within a single input graph?
| /// Throw InputConflictError if the input holds more than one fragment of a logical path. | ||
| void check_no_split_paths(const Input& input) const; | ||
| /// Throw PathLengthMismatchError if a path in the input has a subrange end other than its start plus its sequence length. | ||
| static void check_subrange_matches_length(const Input& input); | ||
| /// Describe `input`'s reference path. Throw ReferencePathError if it does not have | ||
| /// exactly one, or if that path visits its first or last node in reverse. | ||
| ReferenceDescription describe_reference(const Input& input) const; | ||
| /// Group each base path's chunks together and sort them by start position. | ||
| void sort_by_base_path_offset(std::vector<Chunk>& chunks); | ||
| /// Throw ChunkOverlapError if two neighbouring chunks of one base path spell different sequences where they overlap. | ||
| /// The chunks must already be sorted by sort_by_base_path_offset(). | ||
| void check_overlaps_agree(const std::vector<Chunk>& chunks) const; | ||
|
|
||
| /// Throw ChunkOverlapError if any path in the chunk does not start at its reference path's first node. | ||
| void check_all_paths_start_at_reference_first_node(const Chunk& chunk) const; | ||
| /// Throw ChunkOverlapError if any path in the chunk does not end at its reference path's last node. | ||
| void check_all_paths_end_at_reference_last_node(const Chunk& chunk) const; |
There was a problem hiding this comment.
I guess each different check is in its own function so that we can then write unit tests that make sure the check catches the problems it is looking for?
That sort of makes sense, but then we get all these extremely specific check methods. And they're interleaved with other things.
Can't we just have one validation function? Or maybe one for inputs and one for chunks? Maybe they could be methods of inputs and of chunks? Or Input could have a validate() and Chunk could override it? We can still check all the cases and maybe make sure different kinds of problems have appropriate exception types.
| // Group by GroupKey, recording each path's start offset and name. | ||
| map<GroupKey, vector<pair<offset_t, string>>> groups; | ||
| input.graph->for_each_path_handle([&](const path_handle_t& path) { | ||
| groups[group_key_of(*input.graph, path)] | ||
| .emplace_back(path_start_offset(input.graph->get_subrange(path), | ||
| input.graph->get_phase_block(path)), | ||
| input.graph->get_path_name(path)); | ||
| }); |
There was a problem hiding this comment.
Isn't this duplicative with similar passes elsewhere? How much of the validation ends up re-doing work that the actual operation will need to do?
|
I skimmed Adam's reviews and, wait, we have a type called |
|
@faithokamoto I think it's fine because where it's defined it's inside the class that it's an input for. So you address it as It's still a bit of a non-idea abstraction ("What's a |
|
I looked over the auto-review findings, and of those I think we should:
|
Changelog Entry
To be copied to the draft changelog by merger:
vg combinegained options for stitching chunked assemblies and graphs:-f/--connect-fragments(now the default mode),-s/--shared-nodes,-m/--merge, and-u/--fusion.vg concatis removed (use updatedvg combineinstead)Description
This PR adds four new options to
vg combineto support our chunked-assembly stitching workflow.-f/--connect-fragments(default mode): Treats paths that share the same sample/locus/haplotype/phase block but differ only by subrange — e.g.CHM13#0#chr22andCHM13#0#chr22[17475777]— as fragments of one path. It sorts the fragments by start offset, concatenates them in order, adds the missing inter-fragment edges, and rewrites the path name to a single[start-end]subrange covering the union. This runs by default whenever none of-c/-p/-mis given.-s/--shared-nodes: Modifies the default (-f) mode so node IDs are not renumbered when merging. Node IDs appearing in both inputs must carry the same sequence; one copy is kept. This is intended forvg chunkoutput that shares boundary nodes across chunks, where the graph (rather than the assembly) was chunked. Fragments are merged into one path, trimming duplicated boundary steps. Valid only with the default mode; cannot be combined with-c/-p/-m.-m/--merge: Overlap-aware stitching for chunked-assembly graphs that share REFERENCE-sense paths (e.g.CHM13#0#chr22). Inputs are bucketed by REFERENCE identity (sample/locus/haplotype), so chunks of different chromosomes (e.g.CHM13#0#chr21andCHM13#0#chr22) can be passed together — each reference is stitched independently and kept as a separate path in the output. Within a reference, it sorts inputs by REFERENCE start offset, validates that each chunk boundary is a single chain node shared by every path (and equal to the REFERENCE end), trims each right chunk's leading node by the REFERENCE-offset overlap, connects the left end to the trimmed right start, then merges fragments like-f.-u/--fusion(requires-m): Instead of adding an edge between the left end node and the right chunk's trimmed start node, fuses them into a single node carrying the concatenated sequence.